{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "[View in Colaboratory](https://colab.research.google.com/github/theamrzaki/seq2seq_Amazon_Reviews_tf_1.11/blob/master/Seq2Seq2.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MaILF2Y66mjv"
   },
   "source": [
    "# Summarizing Text with Amazon Reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AOgbGhKS6mjx"
   },
   "source": [
    "The objective of this project is to build a model that can create relevant summaries for reviews written about fine foods sold on Amazon. This dataset contains above 500,000 reviews, and is hosted on [Kaggle.](https://www.kaggle.com/snap/amazon-fine-food-reviews)\n",
    "\n",
    "To build our model we will use a two-layered bidirectional RNN with LSTMs on the input data and two layers, each with an LSTM using bahdanau attention on the target data. [Jaemin Cho's tutorial](https://github.com/j-min/tf_tutorial_plus/tree/master/RNN_seq2seq/contrib_seq2seq) for seq2seq was really helpful to get the code in working order because this is my first project with TensorFlow 1.1; some of the functions are very different from 1.0. The architecture for this model is similar to Xin Pan's and Peter Liu's, here's their [GitHub page.](https://github.com/tensorflow/models/tree/master/textsum)\n",
    "\n",
    "The sections of this project are:\n",
    "- Inspecting the Data\n",
    "- Preparing the Data\n",
    "- Building the Model\n",
    "- Training the Model\n",
    "- Making Our Own Summaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Tw0keeCK6mjy",
    "outputId": "75721500-2cde-4e68-c974-cee9e555cd6b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow Version: 1.10.1\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "import time\n",
    "from tensorflow.python.layers.core import Dense\n",
    "from tensorflow.python.ops.rnn_cell_impl import _zero_state_tensors\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "V-9OEfMioyUa",
    "outputId": "296746dc-e4bd-47a5-d886-9363a64a8207"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found GPU at: /device:GPU:0\n"
     ]
    }
   ],
   "source": [
    "device_name = tf.test.gpu_device_name()\n",
    "if device_name != '/device:GPU:0':\n",
    "  raise SystemError('GPU device not found')\n",
    "print('Found GPU at: {}'.format(device_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H0o8PPgD63ZP"
   },
   "source": [
    "### Google Connect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 2708
    },
    "colab_type": "code",
    "id": "frzAZi2D67tn",
    "outputId": "23cca02f-0eae-4113-c91f-2323c7d098d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preconfiguring packages ...\n",
      "Selecting previously unselected package cron.\n",
      "(Reading database ... 18408 files and directories currently installed.)\n",
      "Preparing to unpack .../00-cron_3.0pl1-128ubuntu5_amd64.deb ...\n",
      "Unpacking cron (3.0pl1-128ubuntu5) ...\n",
      "Selecting previously unselected package libapparmor1:amd64.\n",
      "Preparing to unpack .../01-libapparmor1_2.11.0-2ubuntu17.1_amd64.deb ...\n",
      "Unpacking libapparmor1:amd64 (2.11.0-2ubuntu17.1) ...\n",
      "Selecting previously unselected package libdbus-1-3:amd64.\n",
      "Preparing to unpack .../02-libdbus-1-3_1.10.22-1ubuntu1_amd64.deb ...\n",
      "Unpacking libdbus-1-3:amd64 (1.10.22-1ubuntu1) ...\n",
      "Selecting previously unselected package dbus.\n",
      "Preparing to unpack .../03-dbus_1.10.22-1ubuntu1_amd64.deb ...\n",
      "Unpacking dbus (1.10.22-1ubuntu1) ...\n",
      "Selecting previously unselected package dirmngr.\n",
      "Preparing to unpack .../04-dirmngr_2.1.15-1ubuntu8.1_amd64.deb ...\n",
      "Unpacking dirmngr (2.1.15-1ubuntu8.1) ...\n",
      "Selecting previously unselected package distro-info-data.\n",
      "Preparing to unpack .../05-distro-info-data_0.36ubuntu0.2_all.deb ...\n",
      "Unpacking distro-info-data (0.36ubuntu0.2) ...\n",
      "Selecting previously unselected package libkmod2:amd64.\n",
      "Preparing to unpack .../06-libkmod2_24-1ubuntu2_amd64.deb ...\n",
      "Unpacking libkmod2:amd64 (24-1ubuntu2) ...\n",
      "Selecting previously unselected package kmod.\n",
      "Preparing to unpack .../07-kmod_24-1ubuntu2_amd64.deb ...\n",
      "Unpacking kmod (24-1ubuntu2) ...\n",
      "Selecting previously unselected package lsb-release.\n",
      "Preparing to unpack .../08-lsb-release_9.20160110ubuntu5_all.deb ...\n",
      "Unpacking lsb-release (9.20160110ubuntu5) ...\n",
      "Selecting previously unselected package libgirepository-1.0-1:amd64.\n",
      "Preparing to unpack .../09-libgirepository-1.0-1_1.54.1-1_amd64.deb ...\n",
      "Unpacking libgirepository-1.0-1:amd64 (1.54.1-1) ...\n",
      "Selecting previously unselected package gir1.2-glib-2.0:amd64.\n",
      "Preparing to unpack .../10-gir1.2-glib-2.0_1.54.1-1_amd64.deb ...\n",
      "Unpacking gir1.2-glib-2.0:amd64 (1.54.1-1) ...\n",
      "Selecting previously unselected package iso-codes.\n",
      "Preparing to unpack .../11-iso-codes_3.75-1_all.deb ...\n",
      "Unpacking iso-codes (3.75-1) ...\n",
      "Selecting previously unselected package libdbus-glib-1-2:amd64.\n",
      "Preparing to unpack .../12-libdbus-glib-1-2_0.108-2_amd64.deb ...\n",
      "Unpacking libdbus-glib-1-2:amd64 (0.108-2) ...\n",
      "Selecting previously unselected package python-apt-common.\n",
      "Preparing to unpack .../13-python-apt-common_1.4.0~beta3build2_all.deb ...\n",
      "Unpacking python-apt-common (1.4.0~beta3build2) ...\n",
      "Selecting previously unselected package python3-apt.\n",
      "Preparing to unpack .../14-python3-apt_1.4.0~beta3build2_amd64.deb ...\n",
      "Unpacking python3-apt (1.4.0~beta3build2) ...\n",
      "Selecting previously unselected package python3-dbus.\n",
      "Preparing to unpack .../15-python3-dbus_1.2.4-1build3_amd64.deb ...\n",
      "Unpacking python3-dbus (1.2.4-1build3) ...\n",
      "Selecting previously unselected package python3-gi.\n",
      "Preparing to unpack .../16-python3-gi_3.24.1-2build1_amd64.deb ...\n",
      "Unpacking python3-gi (3.24.1-2build1) ...\n",
      "Selecting previously unselected package module-init-tools.\n",
      "Preparing to unpack .../17-module-init-tools_24-1ubuntu2_all.deb ...\n",
      "Unpacking module-init-tools (24-1ubuntu2) ...\n",
      "Selecting previously unselected package python-apt.\n",
      "Preparing to unpack .../18-python-apt_1.4.0~beta3build2_amd64.deb ...\n",
      "Unpacking python-apt (1.4.0~beta3build2) ...\n",
      "Selecting previously unselected package python-pycurl.\n",
      "Preparing to unpack .../19-python-pycurl_7.43.0-2build2_amd64.deb ...\n",
      "Unpacking python-pycurl (7.43.0-2build2) ...\n",
      "Selecting previously unselected package python-software-properties.\n",
      "Preparing to unpack .../20-python-software-properties_0.96.24.17_all.deb ...\n",
      "Unpacking python-software-properties (0.96.24.17) ...\n",
      "Selecting previously unselected package python3-software-properties.\n",
      "Preparing to unpack .../21-python3-software-properties_0.96.24.17_all.deb ...\n",
      "Unpacking python3-software-properties (0.96.24.17) ...\n",
      "Selecting previously unselected package software-properties-common.\n",
      "Preparing to unpack .../22-software-properties-common_0.96.24.17_all.deb ...\n",
      "Unpacking software-properties-common (0.96.24.17) ...\n",
      "Selecting previously unselected package unattended-upgrades.\n",
      "Preparing to unpack .../23-unattended-upgrades_0.98ubuntu1.1_all.deb ...\n",
      "Unpacking unattended-upgrades (0.98ubuntu1.1) ...\n",
      "Setting up python-apt-common (1.4.0~beta3build2) ...\n",
      "Setting up python3-apt (1.4.0~beta3build2) ...\n",
      "Setting up iso-codes (3.75-1) ...\n",
      "Setting up distro-info-data (0.36ubuntu0.2) ...\n",
      "Setting up python-pycurl (7.43.0-2build2) ...\n",
      "Setting up lsb-release (9.20160110ubuntu5) ...\n",
      "Setting up libgirepository-1.0-1:amd64 (1.54.1-1) ...\n",
      "Setting up libkmod2:amd64 (24-1ubuntu2) ...\n",
      "Setting up gir1.2-glib-2.0:amd64 (1.54.1-1) ...\n",
      "Processing triggers for libc-bin (2.26-0ubuntu2.1) ...\n",
      "Setting up libapparmor1:amd64 (2.11.0-2ubuntu17.1) ...\n",
      "Setting up unattended-upgrades (0.98ubuntu1.1) ...\n",
      "\n",
      "Creating config file /etc/apt/apt.conf.d/20auto-upgrades with new version\n",
      "\n",
      "Creating config file /etc/apt/apt.conf.d/50unattended-upgrades with new version\n",
      "invoke-rc.d: could not determine current runlevel\n",
      "invoke-rc.d: policy-rc.d denied execution of start.\n",
      "Setting up dirmngr (2.1.15-1ubuntu8.1) ...\n",
      "Setting up cron (3.0pl1-128ubuntu5) ...\n",
      "Adding group `crontab' (GID 102) ...\n",
      "Done.\n",
      "update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults\n",
      "update-rc.d: warning: stop runlevel arguments (1) do not match cron Default-Stop values (none)\n",
      "invoke-rc.d: could not determine current runlevel\n",
      "invoke-rc.d: policy-rc.d denied execution of start.\n",
      "Setting up libdbus-1-3:amd64 (1.10.22-1ubuntu1) ...\n",
      "Setting up kmod (24-1ubuntu2) ...\n",
      "Setting up libdbus-glib-1-2:amd64 (0.108-2) ...\n",
      "Setting up python3-gi (3.24.1-2build1) ...\n",
      "Setting up module-init-tools (24-1ubuntu2) ...\n",
      "Setting up python3-software-properties (0.96.24.17) ...\n",
      "Setting up dbus (1.10.22-1ubuntu1) ...\n",
      "Setting up python-apt (1.4.0~beta3build2) ...\n",
      "Setting up python3-dbus (1.2.4-1build3) ...\n",
      "Setting up python-software-properties (0.96.24.17) ...\n",
      "Setting up software-properties-common (0.96.24.17) ...\n",
      "Processing triggers for libc-bin (2.26-0ubuntu2.1) ...\n",
      "Processing triggers for dbus (1.10.22-1ubuntu1) ...\n",
      "\n",
      "Redirecting output to ‘wget-log’.\n",
      "Selecting previously unselected package google-drive-ocamlfuse.\n",
      "(Reading database ... 19816 files and directories currently installed.)\n",
      "Preparing to unpack google-drive-ocamlfuse_0.7.0-0ubuntu1_amd64.deb ...\n",
      "Unpacking google-drive-ocamlfuse (0.7.0-0ubuntu1) ...\n",
      "\u001b[1mdpkg:\u001b[0m dependency problems prevent configuration of google-drive-ocamlfuse:\n",
      " google-drive-ocamlfuse depends on libfuse2 (>= 2.8); however:\n",
      "  Package libfuse2 is not installed.\n",
      "\n",
      "\u001b[1mdpkg:\u001b[0m error processing package google-drive-ocamlfuse (--install):\n",
      " dependency problems - leaving unconfigured\n",
      "Errors were encountered while processing:\n",
      " google-drive-ocamlfuse\n",
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "Correcting dependencies... Done\n",
      "The following additional packages will be installed:\n",
      "  libfuse2\n",
      "Suggested packages:\n",
      "  fuse\n",
      "The following NEW packages will be installed:\n",
      "  libfuse2\n",
      "0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\n",
      "1 not fully installed or removed.\n",
      "Need to get 80.9 kB of archives.\n",
      "After this operation, 313 kB of additional disk space will be used.\n",
      "Get:1 http://archive.ubuntu.com/ubuntu artful/main amd64 libfuse2 amd64 2.9.7-1ubuntu1 [80.9 kB]\n",
      "Fetched 80.9 kB in 0s (120 kB/s)\n",
      "Selecting previously unselected package libfuse2:amd64.\n",
      "(Reading database ... 19821 files and directories currently installed.)\n",
      "Preparing to unpack .../libfuse2_2.9.7-1ubuntu1_amd64.deb ...\n",
      "Unpacking libfuse2:amd64 (2.9.7-1ubuntu1) ...\n",
      "Setting up libfuse2:amd64 (2.9.7-1ubuntu1) ...\n",
      "Processing triggers for libc-bin (2.26-0ubuntu2.1) ...\n",
      "Setting up google-drive-ocamlfuse (0.7.0-0ubuntu1) ...\n",
      "Selecting previously unselected package fuse.\n",
      "(Reading database ... 19833 files and directories currently installed.)\n",
      "Preparing to unpack .../fuse_2.9.7-1ubuntu1_amd64.deb ...\n",
      "Unpacking fuse (2.9.7-1ubuntu1) ...\n",
      "Setting up fuse (2.9.7-1ubuntu1) ...\n",
      "Please, open the following URL in a web browser: https://accounts.google.com/o/oauth2/auth?client_id=32555940559.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&response_type=code&access_type=offline&approval_prompt=force\n",
      "··········\n",
      "Please, open the following URL in a web browser: https://accounts.google.com/o/oauth2/auth?client_id=32555940559.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&response_type=code&access_type=offline&approval_prompt=force\n",
      "Please enter the verification code: Access token retrieved correctly.\n"
     ]
    }
   ],
   "source": [
    "#working google drive ya RAB\n",
    "#https://stackoverflow.com/questions/52385655/unable-to-locate-package-google-drive-ocamlfuse-suddenly-stopped-working\n",
    "\n",
    "\n",
    "!apt-get install -y -qq software-properties-common python-software-properties module-init-tools\n",
    "!wget https://launchpad.net/~alessandro-strada/+archive/ubuntu/google-drive-ocamlfuse-beta/+build/15331130/+files/google-drive-ocamlfuse_0.7.0-0ubuntu1_amd64.deb\n",
    "!dpkg -i google-drive-ocamlfuse_0.7.0-0ubuntu1_amd64.deb\n",
    "!apt-get install -f\n",
    "!apt-get -y install -qq fuse\n",
    "from google.colab import auth\n",
    "auth.authenticate_user()\n",
    "from oauth2client.client import GoogleCredentials\n",
    "creds = GoogleCredentials.get_application_default()\n",
    "import getpass\n",
    "!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL\n",
    "vcode = getpass.getpass()\n",
    "!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}\n",
    "\n",
    "!mkdir -p drive\n",
    "!google-drive-ocamlfuse drive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c7s5j-U46mj5"
   },
   "source": [
    "## Insepcting the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "o24fDOcG6mj6"
   },
   "outputs": [],
   "source": [
    "reviews = pd.read_csv(\"drive/Colab Notebooks/Menu/Data/Reviews.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "pv2B397Z6mj8",
    "outputId": "1f36bde6-ad40-44b2-ce0e-95a14e287137"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(568454, 10)"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 652
    },
    "colab_type": "code",
    "id": "8rmt7bTH6mkA",
    "outputId": "f911dc77-53cd-474b-fa4a-2c3ec8d38a69"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>ProductId</th>\n",
       "      <th>UserId</th>\n",
       "      <th>ProfileName</th>\n",
       "      <th>HelpfulnessNumerator</th>\n",
       "      <th>HelpfulnessDenominator</th>\n",
       "      <th>Score</th>\n",
       "      <th>Time</th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>B001E4KFG0</td>\n",
       "      <td>A3SGXH7AUHU8GW</td>\n",
       "      <td>delmartian</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1303862400</td>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>B00813GRG4</td>\n",
       "      <td>A1D87F6ZCVE5NK</td>\n",
       "      <td>dll pa</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1346976000</td>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>B000LQOCH0</td>\n",
       "      <td>ABXLMWJIXXAIN</td>\n",
       "      <td>Natalia Corres \"Natalia Corres\"</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1219017600</td>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>B000UA0QIQ</td>\n",
       "      <td>A395BORC6FGVXV</td>\n",
       "      <td>Karl</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1307923200</td>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>B006K2ZZ7K</td>\n",
       "      <td>A1UQRSCLF8GW1T</td>\n",
       "      <td>Michael D. Bigham \"M. Wassir\"</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1350777600</td>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id   ProductId          UserId                      ProfileName  \\\n",
       "0   1  B001E4KFG0  A3SGXH7AUHU8GW                       delmartian   \n",
       "1   2  B00813GRG4  A1D87F6ZCVE5NK                           dll pa   \n",
       "2   3  B000LQOCH0   ABXLMWJIXXAIN  Natalia Corres \"Natalia Corres\"   \n",
       "3   4  B000UA0QIQ  A395BORC6FGVXV                             Karl   \n",
       "4   5  B006K2ZZ7K  A1UQRSCLF8GW1T    Michael D. Bigham \"M. Wassir\"   \n",
       "\n",
       "   HelpfulnessNumerator  HelpfulnessDenominator  Score        Time  \\\n",
       "0                     1                       1      5  1303862400   \n",
       "1                     0                       0      1  1346976000   \n",
       "2                     1                       1      4  1219017600   \n",
       "3                     3                       3      2  1307923200   \n",
       "4                     0                       0      5  1350777600   \n",
       "\n",
       "                 Summary                                               Text  \n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...  \n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...  \n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...  \n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...  \n",
       "4            Great taffy  Great taffy at a great price.  There was a wid...  "
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "colab_type": "code",
    "id": "NbtLoD046mkD",
    "outputId": "85fdf5b4-5072-4b52-dcbb-cfd9547c1765"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                         0\n",
       "ProductId                  0\n",
       "UserId                     0\n",
       "ProfileName               16\n",
       "HelpfulnessNumerator       0\n",
       "HelpfulnessDenominator     0\n",
       "Score                      0\n",
       "Time                       0\n",
       "Summary                   27\n",
       "Text                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for any nulls values\n",
    "reviews.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m55yoyyc6mkG"
   },
   "outputs": [],
   "source": [
    "# Remove null values and unneeded features\n",
    "reviews = reviews.dropna()\n",
    "reviews = reviews.drop(['Id','ProductId','UserId','ProfileName','HelpfulnessNumerator','HelpfulnessDenominator',\n",
    "                        'Score','Time'], 1)\n",
    "reviews = reviews.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 195
    },
    "colab_type": "code",
    "id": "p8tL54i16mkJ",
    "outputId": "cbcd1156-9aab-4c12-a0e6-ebbe7cb15568"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Summary                                               Text\n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...\n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...\n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...\n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...\n",
       "4            Great taffy  Great taffy at a great price.  There was a wid..."
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 373
    },
    "colab_type": "code",
    "id": "Z054OKaL6mkM",
    "outputId": "bd316ead-15b0-4f58-e050-5047e0bf1a51"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review # 1\n",
      "Good Quality Dog Food\n",
      "I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than  most.\n",
      "\n",
      "Review # 2\n",
      "Not as Advertised\n",
      "Product arrived labeled as Jumbo Salted Peanuts...the peanuts were actually small sized unsalted. Not sure if this was an error or if the vendor intended to represent the product as \"Jumbo\".\n",
      "\n",
      "Review # 3\n",
      "\"Delight\" says it all\n",
      "This is a confection that has been around a few centuries.  It is a light, pillowy citrus gelatin with nuts - in this case Filberts. And it is cut into tiny squares and then liberally coated with powdered sugar.  And it is a tiny mouthful of heaven.  Not too chewy, and very flavorful.  I highly recommend this yummy treat.  If you are familiar with the story of C.S. Lewis' \"The Lion, The Witch, and The Wardrobe\" - this is the treat that seduces Edmund into selling out his Brother and Sisters to the Witch.\n",
      "\n",
      "Review # 4\n",
      "Cough Medicine\n",
      "If you are looking for the secret ingredient in Robitussin I believe I have found it.  I got this in addition to the Root Beer Extract I ordered (which was good) and made some cherry soda.  The flavor is very medicinal.\n",
      "\n",
      "Review # 5\n",
      "Great taffy\n",
      "Great taffy at a great price.  There was a wide assortment of yummy taffy.  Delivery was very quick.  If your a taffy lover, this is a deal.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspecting some of the reviews\n",
    "for i in range(5):\n",
    "    print(\"Review #\",i+1)\n",
    "    print(reviews.Summary[i])\n",
    "    print(reviews.Text[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_4iCevEI6mkP"
   },
   "source": [
    "## Preparing the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Bgki8Rny6mkP"
   },
   "outputs": [],
   "source": [
    "# A list of contractions from http://stackoverflow.com/questions/19790188/expanding-english-language-contractions-in-python\n",
    "contractions = { \n",
    "\"ain't\": \"am not\",\n",
    "\"aren't\": \"are not\",\n",
    "\"can't\": \"cannot\",\n",
    "\"can't've\": \"cannot have\",\n",
    "\"'cause\": \"because\",\n",
    "\"could've\": \"could have\",\n",
    "\"couldn't\": \"could not\",\n",
    "\"couldn't've\": \"could not have\",\n",
    "\"didn't\": \"did not\",\n",
    "\"doesn't\": \"does not\",\n",
    "\"don't\": \"do not\",\n",
    "\"hadn't\": \"had not\",\n",
    "\"hadn't've\": \"had not have\",\n",
    "\"hasn't\": \"has not\",\n",
    "\"haven't\": \"have not\",\n",
    "\"he'd\": \"he would\",\n",
    "\"he'd've\": \"he would have\",\n",
    "\"he'll\": \"he will\",\n",
    "\"he's\": \"he is\",\n",
    "\"how'd\": \"how did\",\n",
    "\"how'll\": \"how will\",\n",
    "\"how's\": \"how is\",\n",
    "\"i'd\": \"i would\",\n",
    "\"i'll\": \"i will\",\n",
    "\"i'm\": \"i am\",\n",
    "\"i've\": \"i have\",\n",
    "\"isn't\": \"is not\",\n",
    "\"it'd\": \"it would\",\n",
    "\"it'll\": \"it will\",\n",
    "\"it's\": \"it is\",\n",
    "\"let's\": \"let us\",\n",
    "\"ma'am\": \"madam\",\n",
    "\"mayn't\": \"may not\",\n",
    "\"might've\": \"might have\",\n",
    "\"mightn't\": \"might not\",\n",
    "\"must've\": \"must have\",\n",
    "\"mustn't\": \"must not\",\n",
    "\"needn't\": \"need not\",\n",
    "\"oughtn't\": \"ought not\",\n",
    "\"shan't\": \"shall not\",\n",
    "\"sha'n't\": \"shall not\",\n",
    "\"she'd\": \"she would\",\n",
    "\"she'll\": \"she will\",\n",
    "\"she's\": \"she is\",\n",
    "\"should've\": \"should have\",\n",
    "\"shouldn't\": \"should not\",\n",
    "\"that'd\": \"that would\",\n",
    "\"that's\": \"that is\",\n",
    "\"there'd\": \"there had\",\n",
    "\"there's\": \"there is\",\n",
    "\"they'd\": \"they would\",\n",
    "\"they'll\": \"they will\",\n",
    "\"they're\": \"they are\",\n",
    "\"they've\": \"they have\",\n",
    "\"wasn't\": \"was not\",\n",
    "\"we'd\": \"we would\",\n",
    "\"we'll\": \"we will\",\n",
    "\"we're\": \"we are\",\n",
    "\"we've\": \"we have\",\n",
    "\"weren't\": \"were not\",\n",
    "\"what'll\": \"what will\",\n",
    "\"what're\": \"what are\",\n",
    "\"what's\": \"what is\",\n",
    "\"what've\": \"what have\",\n",
    "\"where'd\": \"where did\",\n",
    "\"where's\": \"where is\",\n",
    "\"who'll\": \"who will\",\n",
    "\"who's\": \"who is\",\n",
    "\"won't\": \"will not\",\n",
    "\"wouldn't\": \"would not\",\n",
    "\"you'd\": \"you would\",\n",
    "\"you'll\": \"you will\",\n",
    "\"you're\": \"you are\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kXEYjxGE6mkT"
   },
   "outputs": [],
   "source": [
    "def clean_text(text, remove_stopwords = True):\n",
    "    '''Remove unwanted characters, stopwords, and format the text to create fewer nulls word embeddings'''\n",
    "    \n",
    "    # Convert words to lower case\n",
    "    text = text.lower()\n",
    "    \n",
    "    # Replace contractions with their longer forms \n",
    "    if True:\n",
    "        text = text.split()\n",
    "        new_text = []\n",
    "        for word in text:\n",
    "            if word in contractions:\n",
    "                new_text.append(contractions[word])\n",
    "            else:\n",
    "                new_text.append(word)\n",
    "        text = \" \".join(new_text)\n",
    "    \n",
    "    # Format words and remove unwanted characters\n",
    "    text = re.sub(r'https?:\\/\\/.*[\\r\\n]*', '', text, flags=re.MULTILINE)\n",
    "    text = re.sub(r'\\<a href', ' ', text)\n",
    "    text = re.sub(r'&amp;', '', text) \n",
    "    text = re.sub(r'[_\"\\-;%()|+&=*%.,!?:#$@\\[\\]/]', ' ', text)\n",
    "    text = re.sub(r'<br />', ' ', text)\n",
    "    text = re.sub(r'\\'', ' ', text)\n",
    "    \n",
    "    # Optionally, remove stop words\n",
    "    if remove_stopwords:\n",
    "        text = text.split()\n",
    "        stops = set(stopwords.words(\"english\"))\n",
    "        text = [w for w in text if not w in stops]\n",
    "        text = \" \".join(text)\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FbYOStM-6mkV"
   },
   "source": [
    "We will remove the stopwords from the texts because they do not provide much use for training our model. However, we will keep them for our summaries so that they sound more like natural phrases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 84
    },
    "colab_type": "code",
    "id": "da1ofCGn6mkW",
    "outputId": "8177751d-1028-45f1-f463-042bb706ee25"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
      "Summaries are complete.\n",
      "Texts are complete.\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "  \n",
    "# Clean the summaries and texts\n",
    "clean_summaries = []\n",
    "for summary in reviews.Summary:\n",
    "    clean_summaries.append(clean_text(summary, remove_stopwords=False))\n",
    "print(\"Summaries are complete.\")\n",
    "\n",
    "clean_texts = []\n",
    "for text in reviews.Text:\n",
    "    clean_texts.append(clean_text(text))\n",
    "print(\"Texts are complete.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 373
    },
    "colab_type": "code",
    "id": "Qq7WN7ru6mka",
    "outputId": "425334fd-6151-4c54-f39f-1e3fd5e638f1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clean Review # 1\n",
      "good quality dog food\n",
      "bought several vitality canned dog food products found good quality product looks like stew processed meat smells better labrador finicky appreciates product better\n",
      "\n",
      "Clean Review # 2\n",
      "not as advertised\n",
      "product arrived labeled jumbo salted peanuts peanuts actually small sized unsalted sure error vendor intended represent product jumbo\n",
      "\n",
      "Clean Review # 3\n",
      " delight  says it all\n",
      "confection around centuries light pillowy citrus gelatin nuts case filberts cut tiny squares liberally coated powdered sugar tiny mouthful heaven chewy flavorful highly recommend yummy treat familiar story c lewis lion witch wardrobe treat seduces edmund selling brother sisters witch\n",
      "\n",
      "Clean Review # 4\n",
      "cough medicine\n",
      "looking secret ingredient robitussin believe found got addition root beer extract ordered good made cherry soda flavor medicinal\n",
      "\n",
      "Clean Review # 5\n",
      "great taffy\n",
      "great taffy great price wide assortment yummy taffy delivery quick taffy lover deal\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect the cleaned summaries and texts to ensure they have been cleaned well\n",
    "for i in range(5):\n",
    "    print(\"Clean Review #\",i+1)\n",
    "    print(clean_summaries[i])\n",
    "    print(clean_texts[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ov6gNQzH6mkd"
   },
   "outputs": [],
   "source": [
    "def count_words(count_dict, text):\n",
    "    '''Count the number of occurrences of each word in a set of text'''\n",
    "    for sentence in text:\n",
    "        for word in sentence.split():\n",
    "            if word not in count_dict:\n",
    "                count_dict[word] = 1\n",
    "            else:\n",
    "                count_dict[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "EL2TwE406mkh",
    "outputId": "337834cc-8d74-4c18-9baf-0d279ef3fe79"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of Vocabulary: 132884\n"
     ]
    }
   ],
   "source": [
    "# Find the number of times each word was used and the size of the vocabulary\n",
    "word_counts = {}\n",
    "\n",
    "count_words(word_counts, clean_summaries)\n",
    "count_words(word_counts, clean_texts)\n",
    "            \n",
    "print(\"Size of Vocabulary:\", len(word_counts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "paiPI3pR6mko",
    "outputId": "be158e2f-d3f5-40f1-afe0-4bf72285e29a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word embeddings: 484557\n"
     ]
    }
   ],
   "source": [
    "# Load Conceptnet Numberbatch's (CN) embeddings, similar to GloVe, but probably better \n",
    "# (https://github.com/commonsense/conceptnet-numberbatch)\n",
    "embeddings_index = {}\n",
    "with open('drive/Colab Notebooks/Menu/Data/numberbatch-en-17.02.txt', encoding='utf-8') as f:\n",
    "    for line in f:\n",
    "        values = line.split(' ')\n",
    "        word = values[0]\n",
    "        embedding = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = embedding\n",
    "\n",
    "print('Word embeddings:', len(embeddings_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "kQgzpE_q6mkr",
    "outputId": "7352bd68-d2f0-4e7a-d7c6-5f185751e5ed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words missing from CN: 3044\n",
      "Percent of words that are missing from vocabulary: 2.29%\n"
     ]
    }
   ],
   "source": [
    "# Find the number of words that are missing from CN, and are used more than our threshold.\n",
    "missing_words = 0\n",
    "threshold = 20\n",
    "\n",
    "for word, count in word_counts.items():\n",
    "    if count > threshold:\n",
    "        if word not in embeddings_index:\n",
    "            missing_words += 1\n",
    "            \n",
    "missing_ratio = round(missing_words/len(word_counts),4)*100\n",
    "            \n",
    "print(\"Number of words missing from CN:\", missing_words)\n",
    "print(\"Percent of words that are missing from vocabulary: {}%\".format(missing_ratio))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g4m_anQh6mkx"
   },
   "source": [
    "I use a threshold of 20, so that words not in CN can be added to our word_embedding_matrix, but they need to be common enough in the reviews so that the model can understand their meaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "3wR-Uv4F6mky",
    "outputId": "709ff4d0-76ca-4baa-89b4-f227574958ed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of unique words: 132884\n",
      "Number of words we will use: 65469\n",
      "Percent of words we will use: 49.27%\n"
     ]
    }
   ],
   "source": [
    "# Limit the vocab that we will use to words that appear ≥ threshold or are in GloVe\n",
    "\n",
    "#dictionary to convert words to integers\n",
    "vocab_to_int = {} \n",
    "\n",
    "value = 0\n",
    "for word, count in word_counts.items():\n",
    "    if count >= threshold or word in embeddings_index:\n",
    "        vocab_to_int[word] = value\n",
    "        value += 1\n",
    "\n",
    "# Special tokens that will be added to our vocab\n",
    "codes = [\"<UNK>\",\"<PAD>\",\"<EOS>\",\"<GO>\"]   \n",
    "\n",
    "# Add codes to vocab\n",
    "for code in codes:\n",
    "    vocab_to_int[code] = len(vocab_to_int)\n",
    "\n",
    "# Dictionary to convert integers to words\n",
    "int_to_vocab = {}\n",
    "for word, value in vocab_to_int.items():\n",
    "    int_to_vocab[value] = word\n",
    "\n",
    "usage_ratio = round(len(vocab_to_int) / len(word_counts),4)*100\n",
    "\n",
    "print(\"Total number of unique words:\", len(word_counts))\n",
    "print(\"Number of words we will use:\", len(vocab_to_int))\n",
    "print(\"Percent of words we will use: {}%\".format(usage_ratio))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "7sPlq0P06mk4",
    "outputId": "b894bb1e-b3d8-4b09-e072-dd6ca034d573"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65469\n"
     ]
    }
   ],
   "source": [
    "# Need to use 300 for embedding dimensions to match CN's vectors.\n",
    "embedding_dim = 300\n",
    "nb_words = len(vocab_to_int)\n",
    "\n",
    "# Create matrix with default values of zero\n",
    "word_embedding_matrix = np.zeros((nb_words, embedding_dim), dtype=np.float32)\n",
    "for word, i in vocab_to_int.items():\n",
    "    if word in embeddings_index:\n",
    "        word_embedding_matrix[i] = embeddings_index[word]\n",
    "    else:\n",
    "        # If word not in CN, create a random embedding for it\n",
    "        new_embedding = np.array(np.random.uniform(-1.0, 1.0, embedding_dim))\n",
    "        embeddings_index[word] = new_embedding\n",
    "        word_embedding_matrix[i] = new_embedding\n",
    "\n",
    "# Check if value matches len(vocab_to_int)\n",
    "print(len(word_embedding_matrix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lCQ36Smn6mk-"
   },
   "outputs": [],
   "source": [
    "def convert_to_ints(text, word_count, unk_count, eos=False):\n",
    "    '''Convert words in text to an integer.\n",
    "       If word is not in vocab_to_int, use UNK's integer.\n",
    "       Total the number of words and UNKs.\n",
    "       Add EOS token to the end of texts'''\n",
    "    ints = []\n",
    "    for sentence in text:\n",
    "        sentence_ints = []\n",
    "        for word in sentence.split():\n",
    "            word_count += 1\n",
    "            if word in vocab_to_int:\n",
    "                sentence_ints.append(vocab_to_int[word])\n",
    "            else:\n",
    "                sentence_ints.append(vocab_to_int[\"<UNK>\"])\n",
    "                unk_count += 1\n",
    "        if eos:\n",
    "            sentence_ints.append(vocab_to_int[\"<EOS>\"])\n",
    "        ints.append(sentence_ints)\n",
    "    return ints, word_count, unk_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "547FFl3u6mlA",
    "outputId": "5f08aeca-8987-4fe9-f50f-a5221e698818"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of words in headlines: 25679933\n",
      "Total number of UNKs in headlines: 170450\n",
      "Percent of words that are UNK: 0.66%\n"
     ]
    }
   ],
   "source": [
    "# Apply convert_to_ints to clean_summaries and clean_texts\n",
    "word_count = 0\n",
    "unk_count = 0\n",
    "\n",
    "int_summaries, word_count, unk_count = convert_to_ints(clean_summaries, word_count, unk_count)\n",
    "int_texts, word_count, unk_count = convert_to_ints(clean_texts, word_count, unk_count, eos=True)\n",
    "\n",
    "unk_percent = round(unk_count/word_count,4)*100\n",
    "\n",
    "print(\"Total number of words in headlines:\", word_count)\n",
    "print(\"Total number of UNKs in headlines:\", unk_count)\n",
    "print(\"Percent of words that are UNK: {}%\".format(unk_percent))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dnRzSx6M6mlD"
   },
   "outputs": [],
   "source": [
    "def create_lengths(text):\n",
    "    '''Create a data frame of the sentence lengths from a text'''\n",
    "    lengths = []\n",
    "    for sentence in text:\n",
    "        lengths.append(len(sentence))\n",
    "    return pd.DataFrame(lengths, columns=['counts'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 370
    },
    "colab_type": "code",
    "id": "5_i-9rYV6mlF",
    "outputId": "d1abdb07-4ef5-4389-862c-2bd97607032d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summaries:\n",
      "              counts\n",
      "count  568411.000000\n",
      "mean        4.181624\n",
      "std         2.657872\n",
      "min         0.000000\n",
      "25%         2.000000\n",
      "50%         4.000000\n",
      "75%         5.000000\n",
      "max        48.000000\n",
      "\n",
      "Texts:\n",
      "              counts\n",
      "count  568411.000000\n",
      "mean       41.996835\n",
      "std        42.520873\n",
      "min         1.000000\n",
      "25%        18.000000\n",
      "50%        29.000000\n",
      "75%        50.000000\n",
      "max      2085.000000\n"
     ]
    }
   ],
   "source": [
    "lengths_summaries = create_lengths(int_summaries)\n",
    "lengths_texts = create_lengths(int_texts)\n",
    "\n",
    "print(\"Summaries:\")\n",
    "print(lengths_summaries.describe())\n",
    "print()\n",
    "print(\"Texts:\")\n",
    "print(lengths_texts.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "wVMlHmV46mlI",
    "outputId": "89a7a1f4-45fd-483e-eb00-46b41f815afb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "84.0\n",
      "115.0\n",
      "207.0\n"
     ]
    }
   ],
   "source": [
    "# Inspect the length of texts\n",
    "print(np.percentile(lengths_texts.counts, 90))\n",
    "print(np.percentile(lengths_texts.counts, 95))\n",
    "print(np.percentile(lengths_texts.counts, 99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "Ic8buaf16mlL",
    "outputId": "ac1b0cdd-a3d3-4a19-b88a-ebbfbb50c106"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8.0\n",
      "9.0\n",
      "13.0\n"
     ]
    }
   ],
   "source": [
    "# Inspect the length of summaries\n",
    "print(np.percentile(lengths_summaries.counts, 90))\n",
    "print(np.percentile(lengths_summaries.counts, 95))\n",
    "print(np.percentile(lengths_summaries.counts, 99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IPW-kF136mlQ"
   },
   "outputs": [],
   "source": [
    "def unk_counter(sentence):\n",
    "    '''Counts the number of time UNK appears in a sentence.'''\n",
    "    unk_count = 0\n",
    "    for word in sentence:\n",
    "        if word == vocab_to_int[\"<UNK>\"]:\n",
    "            unk_count += 1\n",
    "    return unk_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "EbFt7RTA6mlU",
    "outputId": "c8dec976-e5e4-4586-ff8d-1c2c50eb73e7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "429209\n",
      "429209\n"
     ]
    }
   ],
   "source": [
    "# takes a long time  , this is normal\n",
    "\n",
    "# Sort the summaries and texts by the length of the texts, shortest to longest\n",
    "# Limit the length of summaries and texts based on the min and max ranges.\n",
    "# Remove reviews that include too many UNKs\n",
    "\n",
    "sorted_summaries = []\n",
    "sorted_texts = []\n",
    "max_text_length = 84\n",
    "max_summary_length = 13\n",
    "min_length = 2\n",
    "unk_text_limit = 1\n",
    "unk_summary_limit = 0\n",
    "\n",
    "for length in range(min(lengths_texts.counts), max_text_length): \n",
    "    for count, words in enumerate(int_summaries):\n",
    "        if (len(int_summaries[count]) >= min_length and\n",
    "            len(int_summaries[count]) <= max_summary_length and\n",
    "            len(int_texts[count]) >= min_length and\n",
    "            unk_counter(int_summaries[count]) <= unk_summary_limit and\n",
    "            unk_counter(int_texts[count]) <= unk_text_limit and\n",
    "            length == len(int_texts[count])\n",
    "           ):\n",
    "            sorted_summaries.append(int_summaries[count])\n",
    "            sorted_texts.append(int_texts[count])\n",
    "        \n",
    "# Compare lengths to ensure they match\n",
    "print(len(sorted_summaries))\n",
    "print(len(sorted_texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jYnQFLER6mlY"
   },
   "source": [
    "## Building the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "U1MIZPcl6mlZ"
   },
   "outputs": [],
   "source": [
    "def model_inputs():\n",
    "    '''Create palceholders for inputs to the model'''\n",
    "    \n",
    "    input_data = tf.placeholder(tf.int32, [None, None], name='input')\n",
    "    targets = tf.placeholder(tf.int32, [None, None], name='targets')\n",
    "    lr = tf.placeholder(tf.float32, name='learning_rate')\n",
    "    keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    summary_length = tf.placeholder(tf.int32, (None,), name='summary_length')\n",
    "    max_summary_length = tf.reduce_max(summary_length, name='max_dec_len')\n",
    "    text_length = tf.placeholder(tf.int32, (None,), name='text_length')\n",
    "\n",
    "    return input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dGoxdoHz6mlc"
   },
   "outputs": [],
   "source": [
    "def process_encoding_input(target_data, vocab_to_int, batch_size):\n",
    "    '''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''\n",
    "    \n",
    "    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])\n",
    "    dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)\n",
    "\n",
    "    return dec_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aJ7p3phH6mld"
   },
   "outputs": [],
   "source": [
    "def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):\n",
    "    '''Create the encoding layer'''\n",
    "    \n",
    "    for layer in range(num_layers):\n",
    "        with tf.variable_scope('encoder_{}'.format(layer)):\n",
    "            cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, \n",
    "                                                                    cell_bw, \n",
    "                                                                    rnn_inputs,\n",
    "                                                                    sequence_length,\n",
    "                                                                    dtype=tf.float32)\n",
    "    # Join outputs since we are using a bidirectional RNN\n",
    "    enc_output = tf.concat(enc_output,2)\n",
    "    \n",
    "    return enc_output, enc_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ejCY-IG56mlg"
   },
   "outputs": [],
   "source": [
    "def training_decoding_layer(dec_embed_input, summary_length, dec_cell, initial_state, output_layer, \n",
    "                            vocab_size, max_summary_length):\n",
    "    '''Create the training logits'''\n",
    "    \n",
    "    training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,\n",
    "                                                        sequence_length=summary_length,\n",
    "                                                        time_major=False)\n",
    "\n",
    "    training_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                       training_helper,\n",
    "                                                       initial_state,\n",
    "                                                       output_layer) \n",
    "\n",
    "    training_logits, _ , _ = tf.contrib.seq2seq.dynamic_decode(training_decoder,\n",
    "                                                           output_time_major=False,\n",
    "                                                           impute_finished=True,\n",
    "                                                           maximum_iterations=max_summary_length)\n",
    "    return training_decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p5RpDsV56mll"
   },
   "outputs": [],
   "source": [
    "def inference_decoding_layer(embeddings, start_token, end_token, dec_cell, initial_state, output_layer,\n",
    "                             max_summary_length, batch_size):\n",
    "    '''Create the inference logits'''\n",
    "    \n",
    "    start_tokens = tf.tile(tf.constant([start_token], dtype=tf.int32), [batch_size], name='start_tokens')\n",
    "    \n",
    "    inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(embeddings,\n",
    "                                                                start_tokens,\n",
    "                                                                end_token)\n",
    "                \n",
    "    inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                        inference_helper,\n",
    "                                                        initial_state,\n",
    "                                                        output_layer)\n",
    "                \n",
    "    inference_logits, _ , _ = tf.contrib.seq2seq.dynamic_decode(inference_decoder,\n",
    "                                                            output_time_major=False,\n",
    "                                                            impute_finished=True,\n",
    "                                                            maximum_iterations=max_summary_length)\n",
    "    \n",
    "    return inference_decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MBcDmTXx6mln"
   },
   "outputs": [],
   "source": [
    "def decoding_layer(dec_embed_input, embeddings, enc_output, enc_state, vocab_size, text_length, summary_length, \n",
    "                   max_summary_length, rnn_size, vocab_to_int, keep_prob, batch_size, num_layers):\n",
    "    '''Create the decoding cell and attention for the training and inference decoding layers'''\n",
    "    \n",
    "    for layer in range(num_layers):\n",
    "        with tf.variable_scope('decoder_{}'.format(layer)):\n",
    "            lstm = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                           initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            dec_cell = tf.contrib.rnn.DropoutWrapper(lstm, \n",
    "                                                     input_keep_prob = keep_prob)\n",
    "    \n",
    "    output_layer = Dense(vocab_size,\n",
    "                         kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))\n",
    "    \n",
    "    attn_mech = tf.contrib.seq2seq.BahdanauAttention(rnn_size,\n",
    "                                                  enc_output,\n",
    "                                                  text_length,\n",
    "                                                  normalize=False,\n",
    "                                                  name='BahdanauAttention')\n",
    "\n",
    "    dec_cell = tf.contrib.seq2seq.AttentionWrapper(dec_cell,\n",
    "                                                          attn_mech,\n",
    "                                                          rnn_size)\n",
    "            \n",
    "    #initial_state = tf.contrib.seq2seq.AttentionWrapperState(enc_state[0],\n",
    "    #                                                                _zero_state_tensors(rnn_size, \n",
    "    #                                                                                    batch_size, \n",
    "    #                                                                                    tf.float32)) \n",
    "    initial_state = dec_cell.zero_state(batch_size=batch_size,dtype=tf.float32).clone(cell_state=enc_state[0])\n",
    "\n",
    "    with tf.variable_scope(\"decode\"):\n",
    "        training_decoder = training_decoding_layer(dec_embed_input, \n",
    "                                                  summary_length, \n",
    "                                                  dec_cell, \n",
    "                                                  initial_state,\n",
    "                                                  output_layer,\n",
    "                                                  vocab_size, \n",
    "                                                  max_summary_length)\n",
    "        \n",
    "        training_logits,_ ,_ = tf.contrib.seq2seq.dynamic_decode(training_decoder,\n",
    "                                  output_time_major=False,\n",
    "                                  impute_finished=True,\n",
    "                                  maximum_iterations=max_summary_length)\n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        inference_decoder = inference_decoding_layer(embeddings,  \n",
    "                                                    vocab_to_int['<GO>'], \n",
    "                                                    vocab_to_int['<EOS>'],\n",
    "                                                    dec_cell, \n",
    "                                                    initial_state, \n",
    "                                                    output_layer,\n",
    "                                                    max_summary_length,\n",
    "                                                    batch_size)\n",
    "        \n",
    "        inference_logits,_ ,_ = tf.contrib.seq2seq.dynamic_decode(inference_decoder,\n",
    "                                  output_time_major=False,\n",
    "                                  impute_finished=True,\n",
    "                                  maximum_iterations=max_summary_length)\n",
    "\n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JuVESpih6mlo"
   },
   "outputs": [],
   "source": [
    "def seq2seq_model(input_data, target_data, keep_prob, text_length, summary_length, max_summary_length, \n",
    "                  vocab_size, rnn_size, num_layers, vocab_to_int, batch_size):\n",
    "    '''Use the previous functions to create the training and inference logits'''\n",
    "    \n",
    "    # Use Numberbatch's embeddings and the newly created ones as our embeddings\n",
    "    embeddings = word_embedding_matrix\n",
    "    \n",
    "    enc_embed_input = tf.nn.embedding_lookup(embeddings, input_data)\n",
    "    enc_output, enc_state = encoding_layer(rnn_size, text_length, num_layers, enc_embed_input, keep_prob)\n",
    "    \n",
    "    dec_input = process_encoding_input(target_data, vocab_to_int, batch_size)\n",
    "    dec_embed_input = tf.nn.embedding_lookup(embeddings, dec_input)\n",
    "    \n",
    "    training_logits, inference_logits  = decoding_layer(dec_embed_input, \n",
    "                                                        embeddings,\n",
    "                                                        enc_output,\n",
    "                                                        enc_state, \n",
    "                                                        vocab_size, \n",
    "                                                        text_length, \n",
    "                                                        summary_length, \n",
    "                                                        max_summary_length,\n",
    "                                                        rnn_size, \n",
    "                                                        vocab_to_int, \n",
    "                                                        keep_prob, \n",
    "                                                        batch_size,\n",
    "                                                        num_layers)\n",
    "    \n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "C6w_YGts6mlq"
   },
   "outputs": [],
   "source": [
    "def pad_sentence_batch(sentence_batch):\n",
    "    \"\"\"Pad sentences with <PAD> so that each sentence of a batch has the same length\"\"\"\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [vocab_to_int['<PAD>']] * (max_sentence - len(sentence)) for sentence in sentence_batch]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "72Fvo38MG4kq"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kW6wc5VA6mls"
   },
   "outputs": [],
   "source": [
    "def get_batches(summaries, texts, batch_size):\n",
    "    \"\"\"Batch summaries, texts, and the lengths of their sentences together\"\"\"\n",
    "    for batch_i in range(0, len(texts)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "        summaries_batch = summaries[start_i:start_i + batch_size]\n",
    "        texts_batch = texts[start_i:start_i + batch_size]\n",
    "        pad_summaries_batch = np.array(pad_sentence_batch(summaries_batch))\n",
    "        pad_texts_batch = np.array(pad_sentence_batch(texts_batch))\n",
    "        \n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_summaries_lengths = []\n",
    "        for summary in pad_summaries_batch:\n",
    "            pad_summaries_lengths.append(len(summary))\n",
    "        \n",
    "        pad_texts_lengths = []\n",
    "        for text in pad_texts_batch:\n",
    "            pad_texts_lengths.append(len(text))\n",
    "        \n",
    "        yield pad_summaries_batch, pad_texts_batch, pad_summaries_lengths, pad_texts_lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gUhPBryV6mlv"
   },
   "outputs": [],
   "source": [
    "# Set the Hyperparameters\n",
    "epochs = 100\n",
    "batch_size = 64\n",
    "rnn_size = 256\n",
    "num_layers = 2\n",
    "learning_rate = 0.005\n",
    "keep_probability = 0.75"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 154
    },
    "colab_type": "code",
    "id": "3I3bJqTD6mlx",
    "outputId": "0cd62569-eabc-4dfc-8189-c20ce502159d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn.py:430: calling reverse_sequence (from tensorflow.python.ops.array_ops) with seq_dim is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "seq_dim is deprecated, use seq_axis instead\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py:454: calling reverse_sequence (from tensorflow.python.ops.array_ops) with batch_dim is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "batch_dim is deprecated, use batch_axis instead\n",
      "Graph is built.\n"
     ]
    }
   ],
   "source": [
    "# Build the graph\n",
    "train_graph = tf.Graph()\n",
    "# Set the graph to default to ensure that it is ready for training\n",
    "with train_graph.as_default():\n",
    "    \n",
    "    # Load the model inputs    \n",
    "    input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length = model_inputs()\n",
    "\n",
    "    # Create the training and inference logits\n",
    "    training_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),\n",
    "                                                      targets, \n",
    "                                                      keep_prob,   \n",
    "                                                      text_length,\n",
    "                                                      summary_length,\n",
    "                                                      max_summary_length,\n",
    "                                                      len(vocab_to_int)+1,\n",
    "                                                      rnn_size, \n",
    "                                                      num_layers, \n",
    "                                                      vocab_to_int,\n",
    "                                                      batch_size)\n",
    "    \n",
    "    # Create tensors for the training logits and inference logits\n",
    "    training_logits = tf.identity(training_logits.rnn_output, 'logits')\n",
    "    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')\n",
    "    \n",
    "    # Create the weights for sequence_loss\n",
    "    masks = tf.sequence_mask(summary_length, max_summary_length, dtype=tf.float32, name='masks')\n",
    "\n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            training_logits,\n",
    "            targets,\n",
    "            masks)\n",
    "\n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -5., 5.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)\n",
    "print(\"Graph is built.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fmFOlquE6ml0"
   },
   "source": [
    "## Training the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xprfbevu6ml2"
   },
   "source": [
    "Since I am training this model on my MacBook Pro, it would take me days if I used the whole dataset. For this reason, I am only going to use a subset of the data, so that I can train it over night. Normally I use [FloydHub's](https://www.floydhub.com/) services for my GPU needs, but it would take quite a bit of time to upload the dataset and ConceptNet Numberbatch, so I'm not going to bother with that for this project.\n",
    "\n",
    "I chose not use use the start of the subset because I didn't want to make it too easy for my model. The texts that I am using are closer to the median lengths; I thought this would be more fair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "nlNcRq4j6ml3",
    "outputId": "93e039c3-af63-47fd-df6d-718767a621c0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shortest text length: 25\n",
      "The longest text length: 83\n"
     ]
    }
   ],
   "source": [
    "# Subset the data for training\n",
    "start = 200000\n",
    "end = start + 300000\n",
    "sorted_summaries_short = sorted_summaries[start:end]\n",
    "sorted_texts_short = sorted_texts[start:end]\n",
    "print(\"The shortest text length:\", len(sorted_texts_short[0]))\n",
    "print(\"The longest text length:\",len(sorted_texts_short[-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 11424
    },
    "colab_type": "code",
    "id": "XASfketR6ml6",
    "outputId": "cb708e9b-8fd0-4048-90cb-1128cc01cbca"
   },
   "outputs": [],
   "source": [
    "# Train the Model\n",
    "learning_rate_decay = 0.95\n",
    "min_learning_rate = 0.0005\n",
    "display_step = 20 # Check training loss after every 20 batches\n",
    "stop_early = 0 \n",
    "stop = 6 #3 # If the update loss does not decrease in 3 consecutive update checks, stop training\n",
    "per_epoch = 3 # Make 3 update checks per epoch\n",
    "update_check = (len(sorted_texts_short)//batch_size//per_epoch)-1\n",
    "\n",
    "update_loss = 0 \n",
    "batch_loss = 0\n",
    "summary_update_loss = [] # Record the update losses for saving improvements in the model\n",
    "\n",
    "  \n",
    "tf.reset_default_graph()\n",
    "checkpoint = \"drive/Colab Notebooks/Menu/Model 300/best_model.ckpt\"  #300k sentence\n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # If we want to continue training a previous session\n",
    "    # loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "    # loader.restore(sess, checkpoint)\n",
    "    #sess.run(tf.local_variables_initializer())\n",
    "\n",
    "    for epoch_i in range(1, epochs+1):\n",
    "        update_loss = 0\n",
    "        batch_loss = 0\n",
    "        for batch_i, (summaries_batch, texts_batch, summaries_lengths, texts_lengths) in enumerate(\n",
    "                get_batches(sorted_summaries_short, sorted_texts_short, batch_size)):\n",
    "            start_time = time.time()\n",
    "            _, loss = sess.run(\n",
    "                [train_op, cost],\n",
    "                {input_data: texts_batch,\n",
    "                 targets: summaries_batch,\n",
    "                 lr: learning_rate,\n",
    "                 summary_length: summaries_lengths,\n",
    "                 text_length: texts_lengths,\n",
    "                 keep_prob: keep_probability})\n",
    "\n",
    "            batch_loss += loss\n",
    "            update_loss += loss\n",
    "            end_time = time.time()\n",
    "            batch_time = end_time - start_time\n",
    "\n",
    "            if batch_i % display_step == 0 and batch_i > 0:\n",
    "                print('Epoch {:>3}/{} Batch {:>4}/{} - Loss: {:>6.3f}, Seconds: {:>4.2f}'\n",
    "                      .format(epoch_i,\n",
    "                              epochs, \n",
    "                              batch_i, \n",
    "                              len(sorted_texts_short) // batch_size, \n",
    "                              batch_loss / display_step, \n",
    "                              batch_time*display_step))\n",
    "                batch_loss = 0\n",
    "                \n",
    "                #saver = tf.train.Saver() \n",
    "                #saver.save(sess, checkpoint)\n",
    "                \n",
    "            if batch_i % update_check == 0 and batch_i > 0:\n",
    "                print(\"Average loss for this update:\", round(update_loss/update_check,3))\n",
    "                summary_update_loss.append(update_loss)\n",
    "                \n",
    "              \n",
    "                  \n",
    "                # If the update loss is at a new minimum, save the model\n",
    "                if update_loss <= min(summary_update_loss):\n",
    "                    print('New Record!') \n",
    "                    stop_early = 0\n",
    "                    saver = tf.train.Saver() \n",
    "                    saver.save(sess, checkpoint)\n",
    "\n",
    "                else:\n",
    "                    print(\"No Improvement.\")\n",
    "                    stop_early += 1\n",
    "                    if stop_early == stop:\n",
    "                        break\n",
    "                update_loss = 0\n",
    "            \n",
    "                    \n",
    "        # Reduce learning rate, but not below its minimum value\n",
    "        learning_rate *= learning_rate_decay\n",
    "        if learning_rate < min_learning_rate:\n",
    "            learning_rate = min_learning_rate\n",
    "        \n",
    "        if stop_early == stop:\n",
    "            print(\"Stopping Training.\")\n",
    "            break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S_4rooXoU3lP"
   },
   "outputs": [],
   "source": [
    "checkpoint = \"drive/Colab Notebooks/Menu/Model Backup/best_model.ckpt\" \n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "    loader.restore(sess, checkpoint)\n",
    "    names = []\n",
    "    [names.append(n.name) for n in loaded_graph.as_graph_def().node]\n",
    "names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qLTgl3MT6ml8"
   },
   "source": [
    "## Making Our Own Summaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ycJAqueX6ml9"
   },
   "source": [
    "To see the quality of the summaries that this model can generate, you can either create your own review, or use a review from the dataset. You can set the length of the summary to a fixed value, or use a random value like I have here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gJsdh6eG6ml-"
   },
   "outputs": [],
   "source": [
    "def text_to_seq(text):\n",
    "    '''Prepare the text for the model'''\n",
    "    \n",
    "    text = clean_text(text)\n",
    "    return [vocab_to_int.get(word, vocab_to_int['<UNK>']) for word in text.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 222
    },
    "colab_type": "code",
    "id": "m2Pbp2Cu6mmE",
    "outputId": "e24ae425-7009-4c81-9090-86677e73d187"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from drive/Colab Notebooks/Menu/Model 300/best_model.ckpt\n",
      "Original Text: This is not only a fabulous tea, but I don't get bored with it.<br />The flavor is interesting and robust enough to keep my attention, but not so intense that it's overwhelming.<br />If you generally like rooibos tea but want it with something extra this is a good choice.<br />They somehow managed to put in the perfect amount of lemon: not so much that it's predominant, but not so little that it's undiscernible.\n",
      "Original summary: Really delicious\n",
      "\n",
      "Text\n",
      "  Word Ids:    [456, 174, 406, 6977, 24434, 24453, 68, 1028, 801, 459, 1926, 2685, 2278, 4450, 24434, 24437, 4228, 113, 1125, 174, 662, 304, 716, 0, 1338, 24434, 24572, 8295, 6608, 1962, 200, 1637, 297, 156, 18398, 334, 65465]\n",
      "  Input Words: fabulous tea get bored <br >the flavor interesting robust enough keep attention intense overwhelming <br >if generally like rooibos tea want something extra good choice <br >they somehow managed put perfect amount lemon much predominant little <UNK>\n",
      "\n",
      "Summary\n",
      "  Word Ids:       [30, 90, 174]\n",
      "  Response Words: my favorite tea\n"
     ]
    }
   ],
   "source": [
    "# Create your own review or use one from the dataset\n",
    "#input_sentence = \"I have never eaten an apple before, but this red one was nice. \\\n",
    "                  #I think that I will try a green apple next time.\"\n",
    "#text = text_to_seq(input_sentence)\n",
    "random = np.random.randint(0,len(clean_texts))\n",
    "input_sentence = clean_texts[random]\n",
    "text = text_to_seq(clean_texts[random])\n",
    "\n",
    "checkpoint = \"drive/Colab Notebooks/Menu/Model 300/best_model.ckpt\"\n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "    loader.restore(sess, checkpoint)\n",
    "\n",
    "    input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "    logits = loaded_graph.get_tensor_by_name('predictions:0')\n",
    "    text_length = loaded_graph.get_tensor_by_name('text_length:0')\n",
    "    summary_length = loaded_graph.get_tensor_by_name('summary_length:0')\n",
    "    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "    \n",
    "    #Multiply by batch_size to match the model's input parameters\n",
    "    answer_logits = sess.run(logits, {input_data: [text]*batch_size, \n",
    "                                      summary_length: [np.random.randint(5,8)], \n",
    "                                      text_length: [len(text)]*batch_size,\n",
    "                                      keep_prob: 1.0})[0] \n",
    "\n",
    "# Remove the padding from the tweet\n",
    "pad = vocab_to_int[\"<PAD>\"] \n",
    "\n",
    "print('Original Text:', reviews.Text[random])\n",
    "print('Original summary:', reviews.Summary[random])#clean_summaries[random]\n",
    "\n",
    "print('\\nText')\n",
    "print('  Word Ids:    {}'.format([i for i in text]))\n",
    "print('  Input Words: {}'.format(\" \".join([int_to_vocab[i] for i in text])))\n",
    "\n",
    "print('\\nSummary')\n",
    "print('  Word Ids:       {}'.format([i for i in answer_logits if i != pad]))\n",
    "print('  Response Words: {}'.format(\" \".join([int_to_vocab[i] for i in answer_logits if i != pad])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PMp35dXk6mmI"
   },
   "source": [
    "Examples of reviews and summaries:\n",
    "- Review(1): The coffee tasted great and was at such a good price! I highly recommend this to everyone!\n",
    "- Summary(1): great coffee\n",
    "\n",
    "\n",
    "- Review(2): This is the worst cheese that I have ever bought! I will never buy it again and I hope you won't either!\n",
    "- Summary(2): omg gross gross\n",
    "\n",
    "\n",
    "- Review(3): love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon know quaker flavor packets\n",
    "- Summary(3): love it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UHHJq0pnzDna"
   },
   "source": [
    "**My Examples  (model 150k)**\n",
    "- Review(1): I am not a big fan of cereal but it is nice to have it at the house.  Both my husband and my kid likes this.  I personally prefer protein shake for breakfast.\n",
    "- Original summary(1): Family likes it\n",
    "- My Summary(1): very nice breakfast cereal\n",
    "---\n",
    "- Original Text(2): Good value for a lot of treats! my dog absolutely loves these and it's awesome that they sell them on amazon for half the price of what they would sell it for at a pet store! Will continue buying from here as long as they're available :)\n",
    "- Original summary(2): good value!\n",
    "- My Summary(2): awesome value\n",
    "---\n",
    "- Original Text(3): My baby seems to prefer this formula mixture over the other bigger brands. If you buy it \"Subscribe and Save\" it's a lot cheaper, which is nice. I breastfed for 9 months until my baby stopped gaining weight so we had to start giving her formula. She's been growing steadily ever since!\n",
    "- Original summary(3): Seems healthy, and my baby prefers it\n",
    "- My Summary(3): great formula for baby\n",
    "---\n",
    "- Original Text(4): These are so good, especially when they are still hot.  I love the way they still sizzle and pop while eating.  Sometimes I sprinkle some hot sauce on mine straight from the microwave or a little parmesan cheese, depending on my mood.  However I eat them, they are really good.  Remember to shake them well for a minute or two before nuking them.  The bags do not puff up as much as microwave popcorn, so don't expect them to.  Only cook for the suggested time on the bag.\n",
    "- Original summary(4): Yummmy!\n",
    "- My Summary(4) : good stuff\n",
    "---\n",
    "- Original Text(5): IF YOU WANT A HIGH ENERGY, AND GREAT TASTING (AND I MEAN THE VERY BEST)ENERGY DRINKS, THIS IS THE ONE FOR YOU! BEVNET.COM SAYS IT TASTES LIKE MOUNTAIN DEW, AND IT DOES! VOTED BEST TASTING! HAS JIFFY INGREDIENT THAT MAKES THIS DIFFERENT CALLED D-RIBOSE, AND IT IS A SIMPLE SUGAR THAT IS GREAT FOR ATHLETES! NO BAD THINGS HERE. I HAD A BAD BATCH ONCE, BUT IT WAS A FLUKE.\n",
    "- Original summary(5): THESE ARE THE BEST!\n",
    "- My Summary(5) : the best energy drink\n",
    "---\n",
    "- Original Text(6): What I really appreciate about these bars is that it really takes time to eat and enjoy them. Highly recommended!\n",
    "- Original summary(6): Great product!\n",
    "- My Summary(6) : foolproof boys bars\n",
    "\n",
    "---\n",
    "\n",
    "- Original Text(7): When I was growing up, we ate a fair amount of Spam.  Every so often, I revisit those days, by purchasing some Spam and trying to enjoy it. And I think of the old Monty Python segment focusing on Spam. . . .<br /><br />It's not so easy for me to enjoy! Plain Spam, even cooked, doesn't do it for me.  I start out with good intentions of eating it as part of a meal and normally throw quite a bit of it out. So, this time around, I tried something different--a fried Spam sandwich.<br /><br />I cut up and fried some shallots, slathered a couple pieces of bread with Dijon mustard and catsup, and melted some American cheese over the Spam.  And it worked! This time, I successfully fixed something that didn't feature the Spam taste.  The result was an enjoyable sandwich.<br /><br />So, instead of throwing out all the Spam (as has happened in the past), I'm going to use my imagination this time in preparing this for a meal. In the past, this would have rated 1 to 2 stars; this time around, I'll give it three.\n",
    "- Original summary(7): Back to the future with Spam\n",
    "- Response Words(7): not the best\n",
    "---\n",
    "- Original Text(8): My dog absolutely loves these treats, and the price on these was nearly 1/2 the price I pay at the pet or grocery store!!! Plus I don't have to worry about running out for quite a while, or shall I say my dog does not have to worry!!\n",
    "- Original summary(8): Great deal!\n",
    "- Response Words(8): dog loves these\n",
    "---\n",
    "- Original Text(9): These Ziggies are great to stuff the Kong.  My dog hates his crate, so it keeps him very busy trying to get it out.<br />Sometimes it takes him hours.\n",
    "- Original summary(9): Great item\n",
    "- Response Words(9): dog loves these\n",
    "---\n",
    "- Original Text(10): Got some matcha in Japan from the airport and it was surprisingly tasty.  Still, you are told not to steep green tea for more than a couple minutes or else the bitter flavor comes out.  So, with matcha it's just gonna happen, since you can't take out the tea bag.<br /><br />That said, the Japan matcha was more granular than a \"powder\", more like sand really, and did not produce much bitterness even after ten minutes in hot water.  This Taiwan tea by comparison is a true powder.  It is difficult to dissolve, clumping a quite a bit, and produces a flavor that within a few minutes in hot water becomes bitter, then later even more bitter.<br /><br />UPDATE: It turns out you use a lot less of it than you do of the Japanese matcha I had.  The powder, since it is so fine, goes much further.  To help you with this, they have a spoon in the box to right-size the servings.  I was using a much larger teaspoon instead previously.  Using the smaller amount, it works great for hot tea and does not become bitter.\n",
    "- Original summary(10): Price is OK, flavor a bit bitter\n",
    "- Response Words(10): not as good as i hoped\n",
    "---\n",
    "- Original Tex(11)t: Oh, this is some good cat food!<br />Have you ever opened a can of Friskies and gagged at the awful smell? How can your cat eat that?<br />This cat food smells pretty good, and it simply looks and smells like good food. My kittens made so much noise when I opened that can! They gobbled it right up, and I have to admit that it smelled pretty good.\n",
    "- Original summary(11): Exceptional Cat Food!\n",
    "- Response Words(11): smells good\n",
    "\n",
    "---\n",
    "- Original Text(12): Hmm, at first, I thought this was another one of those beverages competing with Red Bull.  However, as soon as I had my first sip, I knew I was in for something healthier.<br /><br />Hmmm...healthy vs. energy boost?  That is the question of the day.<br /><br />The Switch contains 100% fruit juice and none of the other stuff.  It's just carbonated juice.<br /><br />However, I didn't really quite catch that 'black cherry' flavor.  It tasted more like apple juice than anything else.  Nonetheless, it was still good!\n",
    "- Original summary(12): carbonated juice\n",
    "- Response Words(12): not like cherry\n",
    "---\n",
    "- Original Text: First let me say that these beans taste just fine, however, there are way too many damaged beans in each tin and we were not expecting that.\n",
    "- Original summary: A little disappointed\n",
    "- Response Words: darker beans\n",
    "---\n",
    "- Original Text: The reviews were what sold me on trying this product & the ones I read raved about the taste, so I thought someone should tell the truth.  I'm an avid coffee drinker, so I just wasn't ready for what I found.  It was horrible!!!\n",
    "- Original summary: Horrible taste!!!!!!!\n",
    "-  Response Words: worst coffee i have ever tried so\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "### Model 300k\n",
    "- Original Text: My little Chihuahua and Jack Russell mix just loves these! She gobbles them up! They are a perfect size for her small mouth. According to her, they are very tasty.\n",
    "- Original summary: My Dog Loves Them!\n",
    "- Response Words: perfect size for our retriever\n",
    "--\n",
    "- Original Text: I bought this to serve after I picked some blueberries.  This dessert was a huge hit!  Not to sweet, cooks in 1/2 hour and is good hot or cold.  Also the box is so cute I gave it to my kids teachers as a gift!  They loved it!  Just like homemade.\n",
    "- Original summary: So easy, so good and all natural\n",
    "- Response Words: great hot snack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xko7rtfk8TOR"
   },
   "source": [
    "## Bleu Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "IhAIbFJQ8i7x",
    "outputId": "c1c291e0-36e0-4c92-d21d-c05268503042"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "from nltk.translate.bleu_score import sentence_bleu\n",
    "reference = [['So', 'easy', 'so', 'good','and','all','natural']]\n",
    "candidate = ['great', 'hot', 'snack']\n",
    "score = sentence_bleu(reference, candidate)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6p_CRprLBo2d"
   },
   "outputs": [],
   "source": [
    "!pip install sumeval\n",
    "!python -m spacy download en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 84
    },
    "colab_type": "code",
    "id": "7jmnDRhhBzRL",
    "outputId": "227264b2-c608-4805-de8c-60340c27e4e5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROUGE-1: 0\n",
      "ROUGE-2: 0\n",
      "ROUGE-L: 0\n",
      "ROUGE-BE: 0\n"
     ]
    }
   ],
   "source": [
    "#https://github.com/chakki-works/sumeval\n",
    "#https://github.com/Tian312/awesome-text-summarization\n",
    "\n",
    "from sumeval.metrics.rouge import RougeCalculator\n",
    "\n",
    "refrence_summary = \"So easy, so good and all natural\"\n",
    "model_summary = \"great hot snack\"\n",
    "\n",
    "rouge = RougeCalculator(stopwords=True, lang=\"en\")\n",
    "\n",
    "rouge_1 = rouge.rouge_n(\n",
    "            summary=model_summary,\n",
    "            references=refrence_summary,\n",
    "            n=1)\n",
    "\n",
    "rouge_2 = rouge.rouge_n(\n",
    "            summary=model_summary,\n",
    "            references=[refrence_summary],\n",
    "            n=2)\n",
    "\n",
    "rouge_l = rouge.rouge_l(\n",
    "            summary=model_summary,\n",
    "            references=[refrence_summary])\n",
    "\n",
    "# You need spaCy to calculate ROUGE-BE\n",
    "\n",
    "rouge_be = rouge.rouge_be(\n",
    "            summary=model_summary,\n",
    "            references=[refrence_summary])\n",
    "\n",
    "print(\"ROUGE-1: {}, ROUGE-2: {}, ROUGE-L: {}, ROUGE-BE: {}\".format(\n",
    "    rouge_1, rouge_2, rouge_l, rouge_be\n",
    ").replace(\", \", \"\\n\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 84
    },
    "colab_type": "code",
    "id": "5JMgn0CeKDVD",
    "outputId": "7cc1f4dd-fc4f-4250-b54c-75875e4f5cd4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.29277002 0.        ]\n",
      " [0.         0.        ]]\n",
      "[[0.29277002 0.        ]\n",
      " [0.         0.        ]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "tfidf_vectorizer = TfidfVectorizer()\n",
    "\n",
    "sentence = (\"\"\"My little Chihuahua and Jack Russell mix just loves these! She \n",
    "gobbles them up! They are a perfect\n",
    "size for her small mouth. According\n",
    "to her, they are very tasty.\"\"\",\"\")\n",
    "model = (\"My Dog Loves Them!\",\"\")\n",
    "summary = (\"Chihuahua mix perfect\",\"\")\n",
    "\n",
    "tfidf_matrix_sentence = tfidf_vectorizer.fit_transform(sentence) \n",
    "tfidf_matrix_model = tfidf_vectorizer.transform(model) \n",
    "tfidf_matrix_summary = tfidf_vectorizer.transform(summary) \n",
    "\n",
    "\n",
    "cosine = cosine_similarity(tfidf_matrix_sentence, tfidf_matrix_model)\n",
    "print(cosine)\n",
    "\n",
    "cosine = cosine_similarity(tfidf_matrix_sentence, tfidf_matrix_summary)\n",
    "print(cosine)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IU9ZbhuB6mmJ"
   },
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LGAs1GOr6mmK"
   },
   "source": [
    "I hope that you found this project to be rather interesting and informative. One of my main recommendations for working with this dataset and model is either use a GPU, a subset of the dataset, or plenty of time to train your model. As you might be able to expect, the model will not be able to make good predictions just by seeing many reviews, it needs so see the reviews many times to be able to understand the relationship between words and between descriptions & summaries. \n",
    "\n",
    "In short, I'm pleased with how well this model performs. After creating numerous reviews and checking those from the dataset, I can happily say that most of the generated summaries are appropriate, some of them are great, and some of them make mistakes. I'll try to improve this model and if it gets better, I'll update my GitHub.\n",
    "\n",
    "Thanks for reading!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jFDpGZlk6mmK"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "H0o8PPgD63ZP",
    "IU9ZbhuB6mmJ"
   ],
   "include_colab_link": true,
   "name": "Seq2Seq2.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
